1 link tagged with all of: text to sql + spider benchmark + natural language queries + databases + sql
Links
The article examines why translating natural language into SQL for live data warehouses remains a tough problem. It traces the rise of ad-hoc queries in 1990s‐era historical data stores and shows how schema complexity and performance needs outpace current text-to-SQL systems.
- Benchmarks like Spider (max 200 tables, single database per task) don't reflect real enterprise warehouses with hundreds of tables and thousands of columns spanning dozens of schemas
- Production metadata is messy: cryptic/inconsistent column naming, typos, abbreviations, and mixed-language query logs undermine model performance
- Vendor-specific SQL extensions (Oracle's LISTAGG, PostgreSQL's JSONB operators) and user-defined types routinely break academic text-to-SQL models
- The authors call for shared benchmarks built from anonymized real industry schemas/workloads and argue LLMs need integration with metadata services and human feedback loops before replacing analysts in critical settings
text to sql
spider benchmark
natural language queries
sql
databases